Universal Linear Least - Squares Prediction

An approach to the problem of linear prediction is discussed that is based on recent developments in the universal coding and computational learning theory literature. This development provides a novel perspective on the adaptive filtering problem, and represents a significant departure from traditional adaptive filtering methodologies. In this context, we demonstrate a sequential algorithm for linear prediction whose accumulated squared prediction error, for every possible sequence, is asymptotically as small as the best fixed linear predictor for that sequence. In this work, we consider the problems of adaptive filtering and linear prediction in a competitive algorithm framework. Given a data sequence zn = {x[t]}y=1, the optimal set of N P Ew[N] = x (x [ n ]-x w k z [ n-k ]) 2 , n= 1 k = l is uniquely determined and certainly depends on the input sequence. Recently, a linear prediction algorithm was presented that asymptotically achieves the minimum average sequentially accumulated prediction error over all linear predictors of order p , i.e. min, E, [ N I , for every individual sequence [l]. In this work, we somewhat modify the algorithm, and as a result improve both the algorithm performance, in terms of the bound on the redundancy, and provide a more intuitive proof of this bound. We consider the problem of linear prediction with a filter of fixed-order p , parameterized by the vector w' = [2i)lr. and 6 > 0 is a positive constant. Theorem 1 The total squared prediction error of the p-th-order universal predictor, ln(z, = C y = l (z [ t ]-ZU[t])', satisfies ln(xC,2Cu) 5 mjn { l n (z , 2 G) + 611G112} + A' In 1 1 + Rzz6-' I , Theorem 1 states that the average squared prediction error of the pth-order universal predictor is within O(A2pln(n)/n) of the best batch pth-order linear prediction algorithm, for every individual sequence xn. The idea behind the universal predictor and the proof of the Theorem is as follows. We define a " probability " assignment of each of the continuum of predictors w' E RP to the data sequence xn such that the probability will be an exponentially decreasing function of the total squared-error for that predictor. Over the continuum of predictors with coefficients w', we assign a Gaussian prior over these probabilities, and define the universal probability to be the Bayesian mixture of these probabilities. With the Gaus-sian …